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ABSTRACT 
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decision-making process. Prior to the workshop, a needs assessment including 
trainers, clients, and managers was conducted to determine the framework of 
the program. Instructional objectives were established based on the content 
identified in the needs assessment. Following a review of instructional 
objectives, the workshop addressed assessment techniques in the cognitive, 
affective, and psychomotor domains of learning. Both formative and summative 
evaluations provided feedback on the workshop. An appendix contains the 
faculty needs assessment questionnaire. (Contains 16 references.) 
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Faculty Development: Assessing Learner Achievement 



By 

Barbara A. Frey, D.Ed. 
Karen Overfield, Ed. D. 



Abstract: 



This study addressed the challenges of developing a faculty professional development workshop 
on assessment, measurement, and evaluation of achievement in adult learners. The setting for 
the workshop was a system of post-secondary career colleges throughout the United States. The 
curriculum development model of Kirkpatrick (1994) was used as a tool to guide the decision- 
making process. Prior to the workshop, a needs assessment including trainers, clients, and 
managers was conducted to determine the framework of the program. Instructional objectives 
were established based on the content identified in the needs assessment. Following a review of 
instructional objectives, the workshop addressed assessment techniques in the cognitive, 
affective, and psychomotor domains of learning. Both formative and summative evaluations 
provided feedback on the workshop. 
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Faculty Development: Assessing Learner Achievement 

Evaluation of student learning is a crucial, time-consuming part of an educator’s job. 
Grades assigned by educators have an impact on the personal and professional goals of learners, 
as well as, on their self esteem. The evaluation process too often becomes a means to sort or 
compare learners, rather than a means to enhance their learning. For many educators, assigning 
grades is a subjective process. Some instructors consider effort, attitude, and attendance in the 
grading process. Others focus only on cognitive learning. 

Most educators agree that evaluation should promote learning by focusing on learning 
outcomes and motivating learners to analyze material (Brookfield, 1992; Haladyna, 1999; Kopp, 
1987, Wiggins, 1999). In a literature review, Gronland and Linn (1990) found that students, 
including adult learners, who take courses on a pass/fail basis study less and learn less than when 
they submit themselves to the normal grading process. Furthermore, Foss and Fisher (1988) 
found that assessment strategies significantly influence learning behaviors. For example, 
objective questions about factual material caused learners to limit themselves to memorizing 
information. Higher level cognitive questions motivated them to use higher level thinking skills. 

To be effective and meaningful, the process of evaluation needs to be conducted by 
educators skilled in its application. In order to address the need for knowledge in assessment and 
evaluation of learner achievement in a post-secondary career school, a faculty professional 
development workshop was designed. This article describes the detailed program planning 
process where the adult educators become adult learners. 
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The Director of Faculty Development and a faculty member designed this workshop on 
assessment, measurement, and evaluation of student achievement over a seven-month time 
period. The setting was a career-focused, post-secondary education system consisting of 19 
schools across the United States. Over 1000 faculty members teach in 15 different majors 
including business, technology, and creative fields. Both associate and bachelor degrees are 
offered to an enrollment of 14,000 students. 

Faculty members in this organization are recruited from industry. The system of schools 
takes pride in hiring faculty to teach who are successful and recognized in the field. Instructors 
are also practitioners. They come to the school with a great deal of professional expertise, but 
limited background in teaching. The area of assessment and evaluation presented a challenge to 
them. The goal of this workshop was to provide faculty with the foundation of evaluation and 
assessment knowledge they need to be skilled educators. 

Terminology 

To add to the challenge of evaluation, educational terminology contains several closely 
related words which are often confused and used interchangeably. For this project, key terms 
were defined: 

• Assessment is a multifaceted, comprehensive analysis of performance (Wiggins, 1999). 
Assessments are used to analyze student accomplishment, carefully judging the quality and 
range of achievement. 
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• Evaluation is the systematic process of collecting, analyzing, and interpreting information to 
make an informed decision (Haladyna, 1999). The Stricklands (1998) wrote “evaluation is 
the product of assessment” (p.22). 

• Grading is the process of labeling an evaluation with a letter grade, number, or percentage. 
Grades are summaries of achievement (Haladyna, 1999). 

• Measurement involves the assigning of numbers based on rules to describe some attribute 
Haladyna, 1999). 

• Tests are instruments or measuring devices (Wiggins, 1999). 

Instructional Design 

In planning this faculty workshop, the ADDIE Model was used as a systematic 
framework to guide the development process. Instructional design models (Rothwell & 
Cookson, 1997, p. 195) have many variations, but most include these five stages: 

1 . Analysis of learning needs 

2. Design of training curriculum 

3. Development of training curriculum 

4. Implementation or delivery 

5. Evaluation 

1. Analysis of Learning Needs 

A task force was formed of representatives from several schools throughout the system. 
The task force met on a regular basis for approximately six months as a whole group through 
audio conference calls. The task force formed subgroups to tackle specific areas. These 



subgroups, consisting of faculty, students, and administrators, met face-to-face and reported 
back on progress during scheduled the audio calls. Most of the work of the task force was 
completed in the subgroups. The task force agreed on these goals for the program: 

• Reduce risk of law suits related to student assessment 

• Recognize “excellent” trainers through having them to deliver program 

• Create program format design that was transferable and easy to use 

• Incorporate program into faculty development programs at the school level 

• Provide developmental opportunities for faculty who deliver program 

2. Design of Learning Curriculum 

In planning for the program, the designers wanted to “walk the talk” - role model 
program curriculum design and provide a developmental opportunity for faculty through the 
curriculum design process as well as the workshop. During the design phase, the program 
planners established these objectives to meet the instructional need: 

• Define and explain basic assessment terms as they relate to the classroom 
environment. 

• Identify and determine components of instructional objectives. 

• Determine assessment techniques which support instructional objectives. 

• Compare and contrast assessment techniques for effectiveness in the cognitive, 
affective, and psychomotor domains. 

• Develop techniques to assess student achievement in each of the three learning 
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• Critique instructor feedback comments to identify those that support student learning 
and improvement. 

• Modify course grading system. 

• Compare and contrast traditional letter, point, and percentage systems of determining 
student grades. 

3. Development of Curriculum 



Workshop development represented a time consuming process. First, the program objectives 
were logically organized into this outline. 

I. What are we talking about? 

A. Program overview 

B. Terminology 

1. Measurement 

2. Assessment 

3. Evaluation 

a. Summative 

b. Formative 

II. How do we do it? 

A. Instructional Objectives 

1 . Learning experience 

2. Observable outcomes 

B. Level of mastery 

1. Norm-referenced 

2. Criterion-referenced 

III. How can I assess it? 

A. Knowledge: Cognitive domain 

1. Pre/post-tests 

2. Test construction 

3. Learning contracts 

B. Attitudes: Affective domain 

1 . Rating scales 

2. Case studies 

3. Essays 
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C. Skills: Psychomotor domain 

1 . Self/peer assessments 

2. Critiques 

3. Checklists 

IV. What about grades? 

A. Providing constructive feedback 

B. Grading systems 

1. Conversion 

2. Weighted points 

V. What are the issues? 

A. Grade inflation 

B. Instruction/evaluation improvement 

In addition, program development included: selecting participants, determining the best 
schedule and format, creating a budget, selecting appropriate facilities, selecting appropriate 
trainers, selecting and preparing audiovisual aids, and coordinating the program. Because of the 
close relationship among all of these steps, much of the work on them was done simultaneously. 



4. Implementation 

The program planners agreed on the importance of modeling the principles of adult 
learning in the workshop planning and facilitation. The program was a highly interactive format 
making use of individual, partner, and small group experiential activities. The activities were 
designed to address individual and organizational needs. Pre-program announcements asked 
faculty members to bring their tests and syllabi to the session. In workshop exercises, faculty 
applied the concepts immediately to their materials, and received timely feedback. 

Since there were 1 9 schools and the system did not have someone dedicated as a system 
trainer, a leader’s guide was created as a standardized resource for the facilitators. The format 
was flexible enough for the facilitators to “make it their own” yet structured so they would not 
have to duplicate effort. The leader’s guide contained both an outline of each section of the 
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module as well as a “scripted” version to help the facilitator prepare to teach the module. 
Transparencies and participant handouts were created as well as answer keys and explanations. 
Participant manuals provided the course and content overview, activities, assessment examples, 
related articles, and a resource list. Disks were provided for the facilitators to modify and 
duplicate the documentation. 

Once the design was completed, a pilot program was conducted. The pilot provided an 
opportunity to conduct formative evaluation. The pilot, moreover, allowed the program planners 
to evaluate workshop effectiveness in a clear, timely manner. One of the schools volunteered to 
conduct the pilot as part of their regular faculty in-service program. Minor modifications were 
made to the program resulting from feedback from the pilot study. 

To build a community of learning, it was decided to “certify” instructors in the system as 
facilitators. This process served as a form of reward and recognition for faculty. Schools could 
nominate strong faculty who were interested in the topic of assessment and evaluation. The 
process consisted of participating in the workshop, team teaching it with a “certified” facilitator, 
and then teaching it on their own. Through a coaching and mentoring process new trainers 
gained subject matter expertise. Once “certified” they could teach the module at their own 
school or perhaps be called upon by other schools to deliver the workshop at their location to 
their faculty. 

5. Evaluation 

Kirkpatrick’s (1994) four-level evaluation model (reaction, learning, change, and results) 
is the seminal work for determining the effectiveness of business training programs, and offers 
educators a valuable tool in program evaluation. Kirkpatrick addresses the training beyond 
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learning to job application and benefits to the organization. Table 1 defines and describes the 
indicators for each level of the evaluation model. 
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Tablel : Summary of Kirkpatrick’s Four Level Evaluation Model 



LEVEL 


DEFINITION 


INDICATORS 


1 


Learner reaction 


Workshop evaluation form 


2 


Learning 


Workshop assessment techniques 
Informal evaluation of performance 
through learning activities and 
exercises 


3 


Application 


Class Observations 
Round table discussions 
Mentoring of new faculty 


4 


Results 


Decrease in legal suits filed 



Recommendations 

Several problems and concerns surfaced during the program development. Workshop 
development was challenging because of time restraints on the task force. This was an “added 
on” responsibility for the members of the task force. The priorities of those on the task force for 
the project were different than those of the people facilitating the project. 

Many concerns arose over logistics of scheduling the workshop. Should schools 
schedule during the normal work hours, pay people to attend, bring people in during off hours, 
hire “substitutes” to deliver scheduled classes? Should the workshop be voluntary or 
mandatory? Should schools train everyone at the same time or phase in? 

A role reversal took place. The people who typically did the teaching became the 
learners. They assumed behaviors they complained about in their students. The faculty 
members became the “clients” and adult learners. Some wanted to just sit back and “absorb.” 
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Others felt they knew it all and didn’t need to participate. Others wanted an arena to display 
their expertise. 

The development of the workshop took more time than originally anticipated. The 
amount of time required to design and develop this program was grossly under-estimated. 
Initially, the task force quesstimated three months for development start to finish. Even with two 
designers, it was over a seven-month project. The time for the delivery of the workshop was also 
underestimated. Because of the interactive nature of the workshop, the length of the activities 
and modules were hard to judge. The pilot study conducted was essential to the success of the 
program. It allowed for revisions in content, activities, and time allocations. 

Those trainers who were identified to deliver the program needed to make the program 
their own. Not everyone wanted to use the same activities, the format of the leader’s guide 
appealed and worked for some and not for others, schools could not and did not want to just 
“drop” the program in. 

Throughout the program planning process, there was a shortage of current literature about 
assessment and evaluation of adult learner achievement. In some cases, the program planners 
were forced to use theory from secondary education and adapt it to adult learners. Additional 
research in both quantitative and qualitative aspects of assessment and evaluation of learners in 
higher education would be a valuable contribution to the field of adult education. There is also a 
need for study regarding professional development for educators, and evaluation of professional 
development programs. 

It appears well planned faculty development programs can make a difference. This 
workshop provided faculty with necessary knowledge and awareness in the evaluation of learner 
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achievement. In the future, faculty will be faced with the continued challenges of under- 
prepared students, technological innovations, and increased accountability. Faculty must be 
helped to meet these and many more challenges through lifelong learning and professional 
development. Excellence in higher education is inseparable from the excellence of its faculty. 
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Faculty Needs Assessment Questionnaire 



Assessment, Measurement and Evaluation 



Please answer the following questions on techniques to measure or assess student achievement. 
We will use your feedback to guide us in future faculty development initiatives. We are 
interested in how you are evaluating your students and establishing grading criteria, as well as 
your interest in developing additional techniques. 

1 . Please indicate with a check mark which methods you are currently using and those you would 
like to consider or learn more about. 



GRADING METHODS 


CURRENTLY 

USING 


MORE 

INFO 


N/A 


A. Pretests 








B. Learning Contract 








C. Quizzes 








D. Tests (unit, midterm, final) 








E. Papers (research, term, essay) 








F. Assignments 








G. Projects 








H. Journal 








I. Case Studies 








J. Checklists / Rating Scales 








K. Self-Report Techniques 








L. Student Presentation / Demonstration 








M. Peer Appraisals 








N. Other: 
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2. Indicate with a check mark which of the following criteria you include in determining 
students’ grades. 



CRITERIA 


YES 


NO 


N/A 


A. Effort 








B. Class Participation 








C. Attendance 








D. Attitude 








E. Observations 








F. Spelling 








G. Grammar 








H. Handwriting 








I. Other: 









3. If you give paper and pencil tests, please indicate with a check mark which types of questions 
you currently use or those you would like to consider or learn more about. 



TEST STYLE 


CURRENTLY 

USING 


MORE INFO 


N/A 


A. Multiple Choice 








B. Matching 








C. Essay 








D. Case Study 








E. Completion 








F. Rating Scale/Checklist 








G. True/False 








H. Matching 
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4. If you teach a studio course, please indicate with a check mark those techniques you use to 
assess student achievement. Also check those techniques on which you would like more 
information for future consideration. 



TECHNIQUE 


CURRENTLY 

USING 


MORE INFO 


N/A 


A. Critiques - Individual or Group 








B. Meet Objective / Solve Problem 








C. Use of Type 








D. Color Scheme 








E. Rendering Techniques 








F. Overall Presentation 








G. Neatness / Craftsmanship 








H. Construction Skills 








I. Thumbnail Sketches 








J. Finished Comprehensives 









5. Please answer the following questions based upon the grading procedures of your classes. 



QUESTION 


YES 


NO 


N/A 


A. Do your students know ahead of time what is 
expected to earn each grade? 








B. Is the grading criteria of each course stated on 
your course syllabus? 








C. Are grades students earn in your class an 
accurate indication of what they have learned? 








D. Does every student have an equal opportunity to 
earn each grade? 








E. Do you maintain appropriate documentation and 
accurate records to verify grades? 








F. Are course objectives the basis for rating student 
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achievement? 








G. Is the competence of each student based on pre- 
set standards? 








H. Are evaluation changes or modifications made 
based on a thorough analysis of grading procedures 
and/or policies? 








I. Is the number of projects or tasks completed 
during a quarter considered in your grading 
criteria? 








J. Are your grading policies designed to reduce 
competition among students in the class? 








K. Are grades of a class determined on a curve? 








L. Has “grade inflation” occurred in the grading 
process of your course? 








M. Do you determine grades by converting raw 
scores to a percentage? 








N. Do you provide feedback in the form of 
comments on student tests, projects, or 
assignments? 








0. Are all student projects graded on the same set 
of characteristics? 








P. Does your grading policy include “throwing out” 
the lowest test, quiz, project, or other score? 








Q. Do you give extra credit assignments? 








R. Do you arrange test questions by difficulty, type, 
content, and/or directions? 








S. Do you review individual scored tests as to the 
number of students who missed each question? 








T. Do you use computer graded score sheets? 








U. Do you use a publisher’s test bank which 
accompanies a course textbook? 








V. Do you maintain a test item bank? 
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Please comment on these questions: 



A. Do you grade a finished product, idea, or procedures? Please explain. 



B. Does the amount of time spent on a product/project influence the grade? 



C. On what criteria do you evaluate student projects? How did you determine this criteria? 



D. On what areas of student measurement or grading would you like more information? 



E. Are you confident of your current grading procedures? 
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